Skip to content

Update documentation with 18 changed files (#4474) - #4487

Draft
rysweet wants to merge 3 commits into
mainfrom
feat/issue-4474-ooda-no-progress-breaker-escalation-is-broken-when
Draft

Update documentation with 18 changed files (#4474)#4487
rysweet wants to merge 3 commits into
mainfrom
feat/issue-4474-ooda-no-progress-breaker-escalation-is-broken-when

Conversation

@rysweet

@rysweet rysweet commented Jul 23, 2026

Copy link
Copy Markdown
Owner

Summary

Concise workflow-generated PR for documentation.

Issue

Closes #4474

Changed files

  • docs/concepts/ooda-stuck-label-self-heal.md
  • docs/concepts/reconcile-and-self-deploy.md
  • docs/howto/diagnose-a-no-progress-block.md
  • docs/index.md
  • docs/operations/cognitive-memory-wal-recovery-runbook.md
  • docs/operations/verified-backups.md
  • docs/reference/no-progress-breaker-api.md
  • docs/reference/ooda-stuck-label-self-heal-api.md
  • docs/reference/self-deploy-api.md
  • src/cmd_cleanup/disk.rs
  • src/cmd_cleanup/tests.rs
  • src/journal/mod.rs
  • src/ooda_actions/advance_goal/spawn.rs
  • src/ooda_loop/no_progress.rs
  • src/self_deploy/health.rs
  • src/self_deploy/tests_health.rs
  • src/self_deploy/tests_orchestrator.rs
  • src/stewardship/gh_client.rs

Diff stat

 docs/concepts/ooda-stuck-label-self-heal.md        | 192 +++++++++++
 docs/concepts/reconcile-and-self-deploy.md         |  20 +-
 docs/howto/diagnose-a-no-progress-block.md         |   9 +
 docs/index.md                                      |   2 +
 .../cognitive-memory-wal-recovery-runbook.md       |   7 +-
 docs/operations/verified-backups.md                |  73 ++---
 docs/reference/no-progress-breaker-api.md          |   2 +
 docs/reference/ooda-stuck-label-self-heal-api.md   | 239 ++++++++++++++
 docs/reference/self-deploy-api.md                  |  72 +----
 src/cmd_cleanup/disk.rs                            |  48 +--
 src/cmd_cleanup/tests.rs                           | 138 ++------
 src/journal/mod.rs                                 |   2 +-
 src/ooda_actions/advance_goal/spawn.rs             | 197 ++++++++++--
 src/ooda_loop/no_progress.rs                       |  89 +++++-
 src/self_deploy/health.rs                          | 202 ++----------
 src/self_deploy/tests_health.rs                    | 158 +---------
 src/self_deploy/tests_orchestrator.rs              |   2 -
 src/stewardship/gh_client.rs                       | 351 +++++++++++++++++++++
 18 files changed, 1124 insertions(+), 679 deletions(-)

Behavior

Implemented behavior through these branch commits:
e6e9e31 test(stewardship): fix ETXTBSY flake in ooda-stuck label self-heal tests
b54cc07 refactor(stewardship): collapse single-variant LabelEnsureExecutionError to io::Error
2031589 wip: checkpoint after implementation (steps 7-8)

Validation

ℹ NODE_OPTIONS=--max-old-space-size=32768 (saved preference). To change: /home/azureuser/.amplihack/config
I'll start by understanding the current state — branch, changes made, and toolchain.

● Get branch, status, and diff stat (shell)
│ cd /home/azureuser/src/Simard-deploy-4049/worktrees/feat/issue-4474-ooda-no-progress-breaker-esc
│ alation-is-broken-when && git branch --show-current && echo "---STATUS---" && git status --short
│ && echo "---DIFF STAT---" && git --no-pager diff --stat HEAD
└ 4 lines…

No working-tree changes. Let me check recent commits to see the implemented fix.

● Show recent commits and diff vs main (shell)

Risk

No high-risk subsystem pattern detected from changed paths.

Checklist

  • Branch has 3 commit(s) ahead of main
  • Code review completed
  • Philosophy check passed

This PR was created as a draft for review before merging.


Step 16b: Outside-In Testing Results

Detected toolchain: Rust CLI (Cargo.toml at repo root, edition 2024, crate simard v0.35.0). Package manager: cargo. Escalation surface shells out to the gh CLI. No Node/Python manifests → gadugi-agentic-test not required; per the qa-team Rust-CLI guidance, cargo test is the outside-in harness and the gh subprocess boundary is exercised via stub-gh scripts.

Chosen strategy: (1) Run the crate's real-subprocess tests for the three changed modules (gh_client, no_progress, spawn) — these drive ensure_label through the actual gh subprocess via injected fake-gh scripts. (2) Add an independent end-to-end reproduction with a hand-written stub gh on PATH that reproduces the exact reported journal signature and proves the fixed two-step flow (gh label creategh issue create) escalates successfully across the create / already-exists / unauthorized paths. (3) cargo clippy for CI-green.

Scenarios

# Scenario Boundary / Command Result
Simple Label creatable → attach --label ooda-stuck and file cargo test --lib -- gh_client no_progress spawn (ensure_label_runs_gh_label_create_argv_via_real_executor, ensure_label_attaches_when_create_succeeds) ✅ PASS
Idempotent Label already exists → treated as Attach, no duplicate spam same suite (ensure_label_attaches_idempotently_when_label_already_exists, ensure_label_treats_real_already_exists_stderr_as_attach) ✅ PASS
Edge / degrade gh label create fails (403 / not-found) → degrade to Omit, file issue without label so escalation still proceeds same suite (ensure_label_omits_with_reason_when_unauthorized, ensure_label_degrades_when_real_executor_reports_other_failure, ensure_label_omits_when_spawn_fails_and_never_errs) ✅ PASS
Bug repro (E2E) Stub gh on PATH reproduces could not add label: 'ooda-stuck' not found (old path fails), then fixed two-step flow files the issue across create/already-exists/unauthorized standalone bash harness with stub gh ✅ PASS (old path fails as reported; all three fixed paths escalate)

Key output

  • cargo test --lib -- gh_client no_progress spawn263 passed; 0 failed; 8963 filtered out (46s).
  • cargo clippy --lib --all-featuresFinished, 0 warnings (CI-green).
  • E2E stub-gh harness:
    • Scenario A (no self-heal) → could not add label: 'ooda-stuck' not found → escalation silently fails (reproduces the bug).
    • Scenario B (self-heal) → ✓ Created label ooda-stuck → issue filed → escalation SUCCEEDS.
    • Scenario B2 (already exists) → already exists treated as Attach → issue filed, no duplicate → SUCCEEDS.
    • Scenario C (403 on label create) → disposition Omit, --label dropped → issue still filed → escalation SUCCEEDS.
  • New escalation code emits structured tracing::warn! (targets simard::ooda / simard::ooda_brain) on degrade; no print!/println! added by this PR (verified via git diff origin/main...HEAD).

Fix count: 0. All scenarios passed on the first run; no code changes were required during outside-in testing. The implementation self-heals the ooda-stuck label idempotently (create-if-missing), treats "already exists" as success, and gracefully degrades to filing without the label when the label cannot be ensured — so a Blocked goal is always escalated. Existing one-issue-per-stall dedup is preserved.

rysweet and others added 3 commits July 22, 2026 22:59
Automatic checkpoint to preserve work in progress.
Tests and implementation saved before refactoring phase.
…ror to io::Error

The label self-heal executor uses .output(), which folds spawn+wait into a
single io::Error, so the single-variant LabelEnsureExecutionError enum and its
one-arm formatter label_ensure_execution_reason were pure ceremony (unlike the
multi-stage CreateIssueExecutor whose 3-variant enum carries distinct failure
modes). The injectable fn-pointer seam is preserved; it now returns
Result<Output, io::Error> directly. No behavior change; 24 gh_client / 114
no_progress / 23 spawn tests still pass, clippy + fmt clean.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The three real-executor `ensure_label` tests write a throwaway `gh` script
and immediately exec it via `execute_ensure_label`. Under parallel test
execution a sibling thread's `fork()` (from another `Command::spawn`) can
transiently hold the just-written script's fd open, so the exec races with
ETXTBSY ("text file busy"). `ensure_label_with` maps that `Err` to
`LabelDisposition::Omit`, which non-deterministically breaks the
`already exists -> Attach` assertion (~1-in-3 failures under the multi-threaded
`ensure_label` filter; always green single-threaded).

Absorb the transient at the test-harness layer via a retry-on-ETXTBSY wrapper
(`execute_ensure_label_no_etxtbsy`) rather than complicating the production
executor — a long-installed production `gh` is never written-then-exec'd.

Verified: single-threaded always passed; multi-threaded now 9+/9 green (was
failing ~1-in-3). clippy -D warnings clean, fmt clean.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown

📊 Coverage Summary

Generated by cargo llvm-cov --workspace --summary-only (nightly, excluding test files)

Module Lines Covered Coverage
Total 194780 163748 84.1%

Coverage data from CI run. Test files matching tests?/ are excluded from line counts.

@rysweet rysweet left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Step 17b — Comprehensive Code Review ✅ Approve (non-blocking nits)

Reviewed the full diff for #4474 (label self-heal for the OODA no-progress escalation). The core fix is well-designed, correct, and thoroughly tested. Summary below.

Verdict: Approve. No blocking issues. Two low-severity nits + one accepted edge case.

What's good

  • Correct fail-safe design. ensure_labelLabelDisposition never errors: Attach on success/idempotent-exists, Omit{reason} on any failure, so a blocked goal is always escalated (issue filed with or without the label). This is exactly the requirement.
  • Idempotency. Case-insensitive "already exists" stderr match treats the non-zero "label already present" exit as success. Existing one-issue-per-stall dedup (already_tracked) is untouched.
  • Single source of truth for label argv. LabelDisposition::label_args() + per-site *_issue_argv builders keep --label splicing consistent across all three filer sites.
  • No silent fallbacks. Every degrade path emits structured tracing::warn! (targets simard::ooda / simard::ooda_brain per site); open_tracking_issue was upgraded from .status() (which swallowed non-zero exits) to .output() with a surfaced, bounded failure note — fixing the latent second bug.
  • Injectable seams. LabelEnsureExecutor fn-pointer mirrors the proven CreateIssueExecutor pattern; the three failure classes (create-ok, already-exists, unauthorized, spawn-fail, huge-stderr) are all unit-tested without real gh, plus real-subprocess tests via fake_gh.
  • Hygiene. No new print!/println!, no TODOs/stubs/unimplemented!; all unwrap()/panic! are test-only. Ambient repo scoping (no -R) documented so gh label create and gh issue create can't target different repos.

Nits (non-blocking)

N1 — truncate_for_log is now a third copy. This PR adds gh_client::truncate_for_log (2048 B cap) alongside the pre-existing spawn.rs local (256 B) and ooda_brain::rustyclawd (8 KiB) copies. It's consistent with the repo's existing per-module convention, so not blocking — but note the two escalation failure-note paths now truncate at different bounds: spawn::tracking_issue_failure_note uses the 256 B local helper, while gh_client::ensure_label_with uses the 2048 B one. Worth a future consolidation into one shared helper (e.g. a logfmt util) with a single agreed cap.

N2 — Over-broad visibility. gh_client::truncate_for_log and GH_STDERR_LOG_LIMIT are pub(crate) but only referenced within gh_client. Could be module-private (fn / const) to keep the surface minimal. Trivial.

Accepted edge case (no action needed)

  • TOCTOU between gh label create and gh issue create. If the label were deleted in the sub-second window after ensure_label returns Attach, gh issue create --label would exit non-zero. This is surfaced (not swallowed) via the Ok(out) non-success branches and is astronomically unlikely; documenting here for completeness rather than as a defect.
  • Pre-existing: the no_progress::file_issue Ok(out) error branch logs gh stderr unbounded (stderr = %String::from_utf8_lossy(...)), unlike the new bounded paths. Pre-existing code, out of scope for this PR, but a candidate to route through the same truncation for consistency.

Checklist

  • Code quality and standards — idiomatic, well-documented, matches existing patterns
  • Test coverage adequate — create / already-exists / unauthorized / spawn-fail / huge-stderr / degrade + real-subprocess argv, at all three sites
  • No TODOs, stubs, or swallowed exceptions — .status().output() fix removes the swallow
  • No unimplemented functions
  • Logic correctness — never-error contract verified; idempotent-exists handled
  • Edge case handling — bounded logs, UTF-8-safe truncation, degrade-not-fail; TOCTOU noted as accepted

Build + full suite were green in Step 11b (9219 passed; 0 failed). LGTM to merge.

@rysweet rysweet left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Step 17c — Security Review ✅ No high-confidence vulnerabilities

Threat-modeled the full diff for #4474 (label self-heal for OODA no-progress escalation) across the three source files. The change is defensively designed; no blocking findings.

Checklist

  • Security requirements met — fail-safe degrade preserves escalation; no new privileged surface
  • No new vulnerabilities — see analysis below
  • Sensitive-data handling — no tokens/secrets read, logged, or passed anywhere in the diff
  • AuthN/AuthZ — relies on gh CLI ambient auth; degrades correctly on insufficient scope
  • Injection — argv-only subprocess execution; no shell, no interpolation

Analysis

1. Command / shell injection — SAFE.
Every gh call is Command::new("gh").args(&argv) — a real argv vector, never sh -c/string interpolation (confirmed: no sh -c, /bin/sh, or Command::new("sh") in production diff). title/body are passed as discrete argv elements (--title <title>, --body <body>/--body-file -), so goal-derived content containing shell metacharacters (;, $(), backticks) is delivered literally and never interpreted.

2. Label injection — SAFE.
OODA_STUCK_LABEL is a compile-time constant ("ooda-stuck"). label_args splices ["--label", "ooda-stuck"] unconditionally or nothing — no attacker-controlled label value ever reaches argv.

3. Sensitive-data handling — SAFE.
No GH_TOKEN/GITHUB_TOKEN/.env()/credential references in the diff (confirmed by grep). Auth is entirely delegated to the gh binary's own credential store; the code never touches the token. Logged fields are limited to goal_id, the constant label, truncated gh stderr, title, issue number, and degrade reason — no secret material.

4. Log-flood / DoS — MITIGATED.
GH_STDERR_LOG_LIMIT = 2048 + truncate_for_log bound hostile/runaway gh stderr on a UTF-8 boundary before it is folded into a degrade reason or tracing field. Explicitly tested (label_huge_stderr injects 100 KB → truncated). Good defensive control.

5. AuthZ degrade — SAFE (fail-safe, no priv-esc).
A token with issue-write but not label-write does not fail the escalation: ensure_label returns LabelDisposition::Omit and the issue is still filed (without the label), with the reason surfaced via tracing::warn. Non-zero exits and spawn errors are logged (tracing::error), never silently swallowed — the .status().output() upgrade is the mechanism. No privilege escalation path is introduced.

Non-blocking observations (accepted / out of scope)

  • Log-injection (low): goal-derived title and gh stderr are logged with % formatting. In a plaintext log sink, embedded newlines could forge log lines. Pre-existing pattern, structured tracing fields, low severity — no action required here.
  • TOCTOU (low): a label deleted between ensure_label and gh issue create re-triggers the same missing-label failure, which the design already degrades around safely. Accepted.

Verdict: No high-confidence security findings. argv-only gh invocation, constant label, bounded stderr logging, no secret handling, and fail-safe authz degrade. Approve from a security standpoint.

@rysweet

rysweet commented Jul 23, 2026

Copy link
Copy Markdown
Owner Author

Step 17d — Philosophy Guardian Review ✅ PASS (non-blocking nits)

Assessed the #4474 label self-heal against PHILOSOPHY.md. The change is a textbook fit for the fail-safe / zero-swallow / pragmatic-trust principles.

Compliance checklist

  • Ruthless simplicity achievedLabelDisposition is a 2-variant enum (Attach / Omit{reason}); ensure_label is one subprocess call mapped to that enum. No speculative abstraction, no future-proofing. The executor-injection seam (LabelEnsureExecutor) mirrors the existing CreateIssueExecutor rather than inventing new machinery — consistency over novelty.
  • Bricks & studs followedgh_client.rs remains the single network-touching brick ("the only network-touching surface in this module"). The new studs (ensure_label, LabelDisposition, label_args) are stable pub(crate) contracts; callers (no_progress.rs, spawn.rs) connect only through them. label_args is the deliberate single source of truth for the --label argv fragment across both filer sites. Tests live inside the module.
  • Zero-BS implementation — No stubs / TODOs / unimplemented! / production unwrap (all unwrap are test-only). Crucially, no swallowed exceptions: the .status().output() upgrade surfaces the exit code that was previously silently discarded, and every degrade path emits a structured tracing::warn!/error! with the reason. Failures are visible during development, exactly as required.
  • No over-engineering — The design solves precisely today's need (missing-label breaks escalation) and nothing more. ensure_label intentionally never errors, keeping the caller flow linear.
  • Clean module boundaries — Visibility is pub(crate), keeping the seam internal. The ambient-repo scoping decision (no -R, so label + issue create can't diverge) is documented in code and in docs/concepts/ooda-stuck-label-self-heal.md.

Philosophy highlight — "fail loudly / skip visibly"

This is the standout. PHILOSOPHY.md demands that strict workflow work fail loudly while optional decoration skips visibly. Here the strict work (escalating a blocked goal) always proceeds; the optional label decoration degrades to Omit with a logged, actionable reason instead of silently aborting — which is the exact bug #4474 fixed. Textbook compliance.

Non-blocking nits (consistent with Step 17b)

  • N1: truncate_for_log now exists in 3 modules with inconsistent caps (256 B in spawn.rs vs 2048 B here). Mild DRY tension; matches the pre-existing per-module convention, so acceptable as-is but a future consolidation candidate.
  • N2: truncate_for_log / GH_STDERR_LOG_LIMIT are pub(crate) yet only used in-module — could tighten to private.

Verdict: PASS. No philosophy violations. Nits are cosmetic and non-blocking; build + full suite (9219 passed) already green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant